Papers with captioning models
On the Role of Scene Graphs in Image Captioning (D19-64)
Copied to clipboard
| Challenge: | Recent captioning approaches rely on ad-hoc approaches to obtain graphs for images, but they introduce noise and it is unclear the effect of parser errors on captioning accuracy. |
| Approach: | They investigate whether scene graphs can help image captioning . they show that a scene graph parser can boost performance almost as much as ground truth graphs . |
| Outcome: | The proposed parser can boost performance almost as much as ground truth graphs . |
Combine to Describe: Evaluating Compositional Generalization in Image Captioning (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent work on compositionality has focused on the ability to combine simpler concepts to understand & generate arbitrarily more complex conceptual structures. |
| Approach: | They propose to use a set of image captioning models to benchmark their compositional generalization properties. |
| Outcome: | The proposed models do not generalize in terms of systematicity and productivity, but are robust to synonym substitutions. |
AudioCaps: Generating Captions for Audios in The Wild (N19-1)
Copied to clipboard
| Challenge: | a dataset of 46K audio clips with human-written text pairs is used to generate captions for audio . the task of translating a multimedia input source into natural language has been extensively studied over the past few years . |
| Approach: | They propose a top-down multi-scale encoder and aligned semantic attention for audio captioning. |
| Outcome: | The proposed captions are faithful to audio inputs and better than existing models. |
Enhancing Descriptive Image Captioning with Natural Language Inference (2021.acl-short)
Copied to clipboard
| Challenge: | Existing captioning models tend to generate generic captions, but generating descriptive captions is important. |
| Approach: | They propose a novel approach to encourage captioning models to produce more detailed captions using natural language inference. |
| Outcome: | The proposed method outperforms baseline models on MSCOCO metrics on descriptiveness and descriptiveness. |
How Do Inpainting Artifacts Propagate to Language? (2026.acl-short)
Copied to clipboard
| Challenge: | Figure 1 shows representative examples of visual artifacts introduced by diffusion-based inpainting . despite visually plausible reconstructions, localized inpainding artifactors lead to object substitutions, attribute changes, or category-level errors in downstream captions. |
| Approach: | They propose a diagnostic setup in which masked image regions are reconstructed and then provided to captioning models. |
| Outcome: | The proposed diagnostic framework can be used to examine how visual artifacts affect language generation in vision-language models. |
Content-Specific Humorous Image Captioning Using Incongruity Resolution Chain-of-Thought (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for generating humorous captions are generic and do not capture the content of images. |
| Approach: | They propose a framework that generates content-specific resolutions from fine details extracted from an image and integrates logit bias and negative sampling to suppress the output of generic resolutions. |
| Outcome: | The proposed framework generates humorous captions tailored to the content of specific input images. |
Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag (D19-1)
Copied to clipboard
| Challenge: | Existing models of video captioning use a network and semantics are mixed into one feature. |
| Approach: | They propose an Adaptive Semantic Guidance Network which instantiates whole video semantics to different POS-aware semantics with supervision of part of speech (POS) tag. |
| Outcome: | Extensive experiments show that the proposed model is more efficient than state-of-the-art models. |
VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation (2025.findings-acl)
Copied to clipboard
Xinlong Chen, Yuanxing Zhang, Chongling Rao, Yushuo Guan, Jiaheng Liu, Fuzheng Zhang, Chengru Song, Qiang Liu, Di Zhang, Tieniu Tan
| Challenge: | Existing studies have not identified a link between video caption evaluation and T2V generation. |
| Approach: | They propose a video caption evaluation scheme specifically designed for T2V generation that integrates video annotation with caption evaluation. |
| Outcome: | The proposed system is agnostic to any particular caption format and can be used for training. |
Informative Image Captioning with External Sources of Information (P19-1)
Copied to clipboard
| Challenge: | Current captioning models are trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. |
| Approach: | They propose a mechanism for integrating image information and fine-grained labels into a caption that describes the image in a fluent and informative manner. |
| Outcome: | The proposed model integrates image information with fine-grained labels to produce fluent captions . it can control the appearance of these labels in the output, resulting in fluent and informative captions. |
Models See Hallucinations: Evaluating the Factuality in Video Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics show little correlation with human factuality annotation. |
| Approach: | They propose a weakly-supervised, model-based factuality metric FactVC which outperforms previous metrics on factual evaluation of video captioning. |
| Outcome: | The proposed model outperforms previous metrics on factuality evaluation of video captioning. |
DualFact+: A Multimodal Fact Verification Framework for Procedural Video Captioning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics fail to evaluate factual correctness in procedural video captions . Existing metrics rely on lexical overlap or holistic semantic similarity, but miss role-specific omissions resulting in hallucinations . |
| Approach: | They propose a role-aware, fact-level evaluation framework that distinguishes conceptual facts from contextual facts. |
| Outcome: | Experiments show that state-of-the-art captioning models produce fluent but incomplete descriptions with systematic errors. |